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DESCRIPTION 

Signal Processing Method and Apparatus and Recording Medium 
Technical Field 

This invention relates to a signal processing method and apparatus and to a 
recording medium. More particularly, it relates to a signal processing method and 
apparatus and to a recording medium which takes the difference between the signals 
as detected by the sensor and the real world into account. 

Background Art 

Such a technique is widely exploited which detects events in the real world by 
sensors and which processes sampling data output by the sensors, such as data 
associated with pictures, speech, temperature, pressure, acceleration or odor. 

For example, a picture obtained on imaging an object moving in front of a 
predetermined still background by a video camera is subjected to motion blurring in 
case the object is moved at a higher velocity. 

For example, a picture obtained on imaging an object moving in front of a 
predetermined still background by a video camera employing a CCD is subjected to 
motion blurring in case the object is moved at a higher velocity. That is, when the real 
world is detected by a CCD as a sensor, the picture, as sampling data, undergoes 
distortion. 
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The conventional practice in suppressing this motion blurring is to increase the 
speed of e.g., an electronic shutter to provide for shorter light exposure thne. 

However, inraising the shutter speed in this manner, i,is necessary ,„ adjust the 
^^ioffevideocan^erabe^proceedingtopho^,,^ So )thebIurred 
picture, previously acquired, cannot be corrected to obtain a clear picture. 

^""^hand.ifanobjec.ismovedmrr^^ 
only motion blurring due to mixingof no other than the picmre of the moving object, 
but afeo the mixing of me background picture and the moving object occurs. In the 
conventional system, no consideration is given to detecting the mixing state of the 
background picture and the moving object. 

Moreover, me information of the real world having the space and me time axis 
is acquired by a sensor and made into da*. The data acquired by the sensor is the 
information obtained on projecting the information of the real world in me time and 
space of a lower dimension than the real world So, the information obtained on 
projection is distorted due to projection. Stated differentry, the data output by me 
sensor is distorted relative to the information of the real world. Moreover, th. date, 
distorted by projection, also includes the significant information for correcting the 



distortion. 



In the conventional signal processmg on the sampling data, acquired by the 
semor, the sampling data obtiuoed by the sensor is deemed to be the most reliable 
dam, such that, in subsequent dam processing, such as transmission, recording or 
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reproduction, it has been a sole concern to real** me slate of ^ ^ ^ ^ 

data transmission. 

Heretofore, the sampling data output ^ ^ ^ ^ 

*e sampling data or to ^ ^ rf extractjng ^ 

information buried by projection. 



Disclosure of the Invention 

ItisAereforeanobjectomepre^inventiontoprovideforadjus^entofthe 
amount of motion blurring contained in detection signal of ..blurred picture. 

I. is another object of the present invention to enable detection of anuxingratio 
indicating the S ,a,= of mixing of plural objects such as a background pi«ure and a 

picture of a moving object. 

It is yet another object of the present invention to provide a signal processing 
apparatus in which sampling data oumu, by a sensor may be freed of distortion or the 
significant information can be extract from the sampling da*, for example, to 
provide for adjustment of the amount of motion blurring endued in the detection 
signal if the sampling data is that of picture. 

The present invention proves a picture processing apparatus for processir* 
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device having a predetermined number of pixels each having an integrating effect, the 
picture processing apparatus including processing unit decision means for deciding, 
based on area information specifying a foreground area made up only of foreground 
object components making up a foreground object in the picture data, a background 
area made up only of background object components making up a background object 
in the picture data, and a mixed area which is a mixture of the foreground object 
components and the background object components in the picture data, the mixed area 
including a covered background area formed at a leading end in a movement direction 
of the foreground object, and an uncovered background area formed at a trailing end 
of the foreground object, a processing unit made up of pixel data lying on at least a 
straight line extending in a direction coincident with the direction of movement of the 
foreground object from an outer end of the covered background area to an outer end 
of the uncovered background area, centered about the foreground area, normal 
equation generating means for generating a normal equation by setting pixel values of 
pixels in the processing unit decided based on the processing unit and a dividing value 
which is an unknown dividing value obtained on dividing the foreground object 
components in the mixed area with a predetermined dividing number, and calculating 
means for solving the normal equation by the least square method to generate 
foreground object components adjusted for the quantity of movement blurring. 

The present invention also provides a picture processing method for processing 
picture data made up of a predetermined number of pixel data acquired by an imaging 



PAGE 5/51 * RCVD AT 7(16/2009 1:02:21 PM [Eastern Daytlght Time] * SVR:U8PTO-EFXRF-6/1S * DNIS: 2738300 • C8ID:212S880500 * DURATION (mm-ss): 08-30 



07/16/2009 13:03 FAX 2125880500 FROMMER LAWRENCE & HAUG ©006/051 



5 

device having a predetermined number of pixels each having an integrating effect, the 
picture processing method including a processing unit decision step of deciding, based 
on area information specifying a foreground area made up only of foreground object 
components making up a foreground object in the picture data, a background area 
made up only of background object components making up a background object in the 
picture data, and amixed area which is a mixture of the foreground object components 
and the background object components in the picture data, the mixed area including 
a covered background area formed at a leading end in a movement direction of the 
foreground object, and an uncovered background area formed at a trailing end of the 
foreground object, a processing unit made up of pixel data lying on at least a straight 
line extending in a direction coincident with the direction of movement of the 
foreground object from an outer end of the covered background area to an outer end 
of the uncovered background area, centered about the foreground area, a normal 
equation generating step of generating a normal equation by setting pixel values of 
pixels in the processing unit decided based on the processing unit and a dividing value 
which is an unknown dividing value obtained on dividing the foreground object 
components in the mixed area with a predetermined dividing number, and a calculating 
step of solving the normal equation by the least square method to generate foreground 
object components adjusted for the quantity of movement blurring. 

The present invention also provides a picture processing program for processing 
picture data made up of a predetermined number of pixel data acquired by an imaging 
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device having a predetermined number of pixels each having an integrating effect, the 
picture processing program including a processing unit decision step of deciding, 
based on area information specifying a foreground area made up only of foreground 
object components making up a foreground object in the picture data, a background 
area made up only of background object components making up a background object 
m the picture data, and a mixed area which is a mixture of the foreground object 
components and the background object components in the picture data, the mixed area 
including a covered background area formed at a leading end in a movement direction 
of the foreground object, and an uncovered background area formed at a trailing end 
of the foreground object, a processing unit made up of pixel data lying on at least a 
straight line extending in a direction coincident with the direction of movement of the 
foreground object from an outer end of the covered background area to an outer end 
of the uncovered background area, centered about the foreground area, a normal 
equation generating step of generating a normal equation by setting pixel values of 
pixels in the processing unit decided based on the processing unit and a dividing value 
which is an unknown dividing value obtained on dividing the foreground object 
components in the mixed area with a predetermined dividing number and a calculating 
step of solving the normal equation by the least square method to generate foreground 
object components adjusted for the quantity of movement blurring. 

The present invention also provides a signal processing apparatus for processing 
detection data, acquired every predetermined time period by a sensor made up of a 
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predetermined number of detection elements having time-integrating effects, every 
predetermined time period, the signal processing apparatus including foreground 
sample data extracting means for extracting the sample data present in detection data 
before or after considered detection data where there exists considered sample data 
which is the sample data under consideration, as foreground sample data 
corresponding to an object proving the foreground in the real world, background 
sample data extracting means for extracting the sample data present in detection data 
lying after or before the considered detection data where there exists considered 
sample data which is the sample data under consideration, as background sample data 
corresponding to an object proving the background in the real world and detection 
means for detecting a mixing ratio of the considered sample data based on the 
considered sample data, the foreground sample data and the background sample data. 

The present invention also provides a signal processing method for processing 
detection data, acquired every predetermined time period by a sensor made up of a 
predetermined number of detection elements having time-integrating effects, every 
predetermined time period, the signal processing method including a foreground 
sample data extracting step of extracting the sample data present in detection data 
before or after considered detection data where there exists considered sample data 
which is the sample data under consideration, as foreground sample data 
corresponding to an object proving the foreground in the real world, a background 
sample data extracting step of extracting the sample data present in detection data 
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lying after or before the considered detection data where there exists considered 
sample data which is the sample data under consideration, as background sample data 
corresponding to an object proving the background in the real world and a detection 
step of detecting a mixingratio of the considered sample data based on the considered 
sample data, the foreground sample data and the background sample data. 

The present invention also provides a signal processing program for processing 
detection data, acquired every predetermined time period by a sensor made up of a 
predetermined number of detection elements having time-integrating effects, every 
predetermined time period, the signal processing program including a foreground 
sample data extracting step of extracting the sample data present in detection data 
before or after considered detection data where there exists considered sample data 
which is the sample data under consideration;' as foreground sample data 
corresponding to an object proving the foreground in the real world, a background 
sample data extracting step of extracting the sample data present in detection data 
lying after or before the considered detection data where there exists considered 
sample data which is the sample data under consideration, as background sample data 
corresponding to an object proving the background in th&real world and a detection 
step of detecting a mixing ratio of the considered sample data based on the considered 
sample data, the foreground sample data and the background sample data. 

The present invention also provides a signal processing apparatus for processing 
detection data, acquired every predetermined time period by a sensor made up of a 
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predetermined number of detection elements having time-integrating effects, every 
predetermined time period, the signal processing apparatus including still/movement 
decision means for deciding still/movement based on the detection data, and detection 
means for detecting a mixed area containing sample data having plural real world 
objects mixed together based on the results of discrimination. 

The present invention also provides a signal processing method for processing 
detection data, acquired every predetermined time period by a sensor made up of a 
predetermined number of detection elements having time-integrating effects, every 
predetermined time period, the signal processing method including a still/movement 
decision step of deciding still/movement based on the detection data, and a detection 
step of detecting a mixed area containing sample data having plural real world objects 
mixed together based on the results of discrimination. 

The present invention also provides a signal processing program for processing 
detection data, acquired every predetermined time period by a sensor made up of a 
predetermined number of detection elements having time-integrating effects, every 
predetermined time period, the signal processing program including a still/movement 
decision step of deciding still/movement based on the detection data, and a detection 
step of detecting a mixed area containing sample data having plural real world objects 
mixed together based on the results of discrimination. 

The present invention also provides a signal processing apparatus including 
means for acquiring second signals of a second dimension by projecting first signals 
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as real-world signals of a first dimension on a sensor and by detecting the mapped 
signals by the sensor, the first dimension being lower than the first dimension, and 
signal processing means for extracting the significant information, buried by the 
proj ection from the second signals, by performing signal processing which is based on 

the second signals. 

The present invention also provides a recording medium having recorded 
thereon a computer-readable program, the program including a signal acquisition step 
of acquiring a second signal by projecting a first signal as a real world signal of a first 
dimension on a sensor and detecting the so-mapped first signal by the sensor, the 
signal being of a second dimension lower than the first dimension, and a signal 
processing step of performing signal processing based on the second signal to extract 
the significant information buried by projection from the second signal. 

The present invention provides a signal processing apparatus including signal 
acquisition means for acquiring a second signal by detecting a first signal as a real 
world signal of a first dimension by a sensor, the signal being of a second dimension 
lower than the first dimension and containing distortion with respect to the first signal, 
and signal processing means for performing signal processing on the second signal for 
generating a third signal alleviated in distortion as compared to the second signal. 

The present invention also provides a signal processing apparatus forprocessing 
a predetermined number of detection signals acquired by a sensor made up of a 
predetermined numberof detection elements havmgtimemtegratmg effects, the signal 
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processingapparatusincludingarea specifying means for specifying a foreground area 
made up only of foreground object components constituting an foreground object, a 
background area made up only of background object components constituting a 
background object, and a mixed area mixed from the foreground object components 
and the background object components, mixed area detection means for detecting a 
mixing ratio of the foreground object components and the background object 
components at least in the mixed area, and separating means for separating the 
foreground object and the background object from each other based on the specified 
results by the area specifying means and the mixing ratio. 

The present invention also provides a signal processing method for processing 
a predetermined number of detection signals acquired by a sensor made up of a 
predetermined number of detection elements having time integrating effects, the signal 
processing method including an area specifying step of specifying a foreground area, 
made up only of foreground object components constituting an foreground object, a 
background area made up only of background object components constituting a 
background object, and a mixed area mixed from the foreground object components 
and the background object components, a mixed area detection step of detecting a 
mixing ratio of the foreground object components and the background object 
components at least in the mixed area, and a separating step of separating the 
foreground object and the background object from each other based on the specified 
results by the area specifying means and the mixing ratio. 
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The present invention also provides a recording medium having a computer- 
readable program, recorded thereon, the computer-readable program includingan area 
specifying step of specifying a foreground area, made up only of foreground object 
components constituting an foreground object, a background area made up only of 
background object components constituting a background object, and a mixed area 
mixed from the foreground object components and the background object components, 
a mixed area detection step of detecting a mixing ratio of the foreground object 
components and the background object components at least in the mixed area and a 
separating step of separating the foreground object and the background object from 
each other based on the specified results by the area specifying means and the mixing 



ratio. 



The present invention also provides a signal processing apparatus forprocessing 
a predetermined number of detection signals acquired by a sensor made up of a 
predeterminednumber of detection elements havingtime integrating effects, the signal 
processing apparatus including area specifying means for specifying a foreground area, 
made up only of foreground object components constituting an foreground object, a 
background area made up only of background object components constituting a 
background object, and a mixed area mixed from the foreground object components 
and the background object components, and mixing ratio detecting means for detecting 
a mixing ratio between the foreground object components and the background object 
components at least in the mixed area based on the results specified by the area 
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specifying means. 

The present invention also provides a signal processing method for processing 
a predetermined number of detection signals acquired by a sensor made up of a 
predetennmednumberofdetectionete^^ 

processingmethod including an area specifying step of specifying a foreground area, 
made up only of foreground object components constituting an foreground object, a 
background area made up only of background object components constituting a 
background object, and a mixed area mixed from the foreground object components 
and the background object components, and a mixing ratio detecting step of detecting 
a mixing ratio between the foreground object components and the background object 
components at least in the mixed area based on the results specified by the area 
specifying means. 

The present invention also provides a recording medium having a computer- 
readable program recorded thereon, the signal processing method for processing a 
predetermined number of detection signals acquired by a sensor made up of a 
predetermined number of detection elements having time integrating effects, the 
computer-readable program including an area specifying step of specifying a 
foreground area, made up only of foreground object components constituting an 
foreground object, a background area made up only ofbackground object components 
constituting a background object, and a mixed area mixed from the foreground object 
components and the background object component, and a mixing ratio detecting step 
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of detecting a mixing ratio between the foreground object components and the 
background object components at least n the mixed area based on the results specified 
by uie area specifying means. 

The present invention also provides a signal processing apparatus forprocessing 
a predetermined number of detection signals acquired by a sensor made up of a 
predeterminednumberof detection elements having time integrating effects, the signal 
processing apparatus including mixing ratio detecting means for detecting a mixing 
ratio of foreground object components and background object components in a mixed 
area in which said foreground object components constituting a foreground object and 
said background object components constituting a background object are mixed, and 
separating means for separating the foreground object and the background object from 
each other based on the mixing ratio. 

The present invention provides a signal processing method for processing a 
predetermined number of detection signals acquired by a sensor made up of a 
predetermined number of detection elements havmgtimemtegratmg effects, the signal 
processing method including a mixing ratio detecting step of detecting a mixing ratio 
of foreground object components and background object components in a mixed area 
in which said foreground object components constituting a foreground object and said 
background object components constituting a background object are mixed, and a 
separating step of separating the foreground object and the background object from 
each other based on the mixing ratio. 
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The present invention also provides a recording medium having recorded 
thereon a computer-readable program for processing a predetermined number of 
detection signals acquired by a sensor made up of a predetermined number of 
detection elements having time integrating effects, the computer-readable program 
including a mixing ratio detecting step of detecting a mixing ratio of foreground 
object components and background object components in a mixed area in which said 
foreground object components constituting a foreground object and said background 
object components constituting a background object, and a separating step of 
separating the foreground object and the background object from each other based on 
the mixing ratio. 

Brief Description of the Drawings 

Fig. 1 illustrates the principle of the present invention. 

Fig.2 is a block diagram showing an illustrative structure of a system embodying 
the present invention. 

Fig.3 is a block diagram showing an illustrative structure of a signal processor 
ofFig.2. 

Fig.4 is a flowchart for illustrating the operation of the system of Fig.2. 
Fig.5 illustrates a typical picture acquired at step SI of Fig.4. 
Fig.6 illustrates pixel values of a mixed area. 

Fig.7 illustrates the result of subtracting picture components of the background 
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in domains Dl to D3 of Fig.6. 

Fig.8 illustrates the structure of motion blurring. 

Fig.9 is a flowchart for illustrating another typical processing of the system of 

Fig.2. 

Fig. 10 is a block diagram showing a signal processor 12. 

Fig. 1 1 illustrates the photographing by a sensor. 

Fig. 12 illustrates pixel arrangement 

Fig. 13 illustrates the operation of a detection device. 

Fig. 14 illustrates a picture obtained on imaging an object corresponding to the 
moving foreground and an object coiTesponding to a stationary background 

Fig. 1 5 illustrates a background area, a foreground area, a mixed area, a covered 
background area and an uncovered background area.. 

Fig. 16 is a diagrammatic view showing pixel values of pixels arranged in a 
neighboring relation to another in a row in a picture obtained on imaging an object 
conresponding to the stationary foreground and a picture obtained on imaging an 
object corresponding to the stationary background, with the pixel values extended in 
the time axis direction. 

Fig. 1 7 is a diagrammatic view showing pixel values extended in the time axis 
direction, with the time period corresponding to the shutter time shown split. 

Fig. 18 is a diagrammatic view showing pixel values extended in the time axis 
direction, with the time period corresponding to the shutter time shown split. 
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Fig. 19 is a diagrammatic view showing pixel values extended in the time axis 
direction, with the time period corresponding to the shutter time shown split. 

Fig.20 shows extracted pixels of a foreground area, a background area and a 
mixed area. 

Fig.21 shows the relation of correspondence between pixels and a model 
obtained on expanding the pixel values in the time axis direction. 

Fig.22 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig.23 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig.24 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig.25 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig.26 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig.27 is a flowchart for illustrating the processing for adjusting the amount of 
the motion blurring. 

Fig.28 is a block diagram showing an illustrative structure of an area specifying 
unit 103. 

Fig.29 illustrates apicture as an object corresponding to the foreground is being 



PAGE 18/51 * RCVD AT 7/16/2009 1:02:21 PM [Eastern Daylight Time] * SVR:USPTO-6FXRF-6/15 * DNIS:27383M ' CSID:212S880300 • DURATION (mm-ss):OS-30 



07/16/2009 13:05 FAX 2125880500 FROMMER LAWRENCE & HAUG lg] 019/051 

18 

moved. 

Fig.30 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig. 3 1 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig. 32 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig.33 illustrates a condition for areal decision. 

Figs.34A, 34B, 34C and 34D illustrate the results of identification of areas of 
the area specifying unit 103. 

Fig.35 illustrates the results of identification of areas of the area specifying unit 

103. 

Fig.36 is a flowchart for illustrating the processing for areal identification. 
Fig. 3 7 is a block diagram showing an illustrative structure of a mixing ratio 
calculating unit 104. 

Fig.38 shows a typical ideal mixing ratio a. 

Fig.39 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig.40 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig.4 1 illustrates the approximation exploiting the correlation of the foreground 
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components. 

Fig.42 illustrates the relation between C, N and P. 

Fig.43 is a block diagram showing the structure of an estimated mixing ratio 
processor 201. 

Fig.44 shows a typical estimated mixing ratio. 

Fig.45 is a block diagram showing a modified structure of the mixing ratio 
calculating unit 104 . 

Fig.46 is a flowchart for illustrating the processing for calculating the estimated 
mixing ratio. 

Fig.47 is a flowchart for illustrating the processing for the operation of the 
estimated mixing ratio. t 

Fig.48 is a block diagram showing an illustrative structure of a 
foreground/background separating unit 105. 

Figs.49A and 49B show an input picture, a foreground component picture and 
a background component picture. 

Fig.50 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig.5 1 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig.52 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 
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Fig.53 is a block diagram showing an illustrative structure of a separator 251. 
Figs.54 Aand 54B illustrate typical examples of a foreground component picture 
and a background component picture as separated from each other. 

Fig.55 isaflowchart formustratmgme processing for separatinguie foreground 
and the background from each other. 

Fig.56 is a block diagram showing an illustrative structure of a motion blurring 
adjustment unit 106. 

Fig. 57 illustrating a processing unit. 

Fig.58 is a diagrammatic view showing pixel values of a foreground component 
picture developed in the time axis direction and showing the time period corresponding 
to the shutter period shown split. 

Fig. 59 is a diagrammatic view showing pixel values of a foreground component 
picture developed in the time axis direction and showing the time period corresponding 
to the shutter period shown split. 

Fig. 60 is a diagrammatic view showing pixel values of a foreground component 
picture developed in the time axis direction and showing the time period corresponding 
to the shutter period shown split. 

Fig.61 is a diagrammatic view showing pixel values ofa foreground component 
picture developed in the time axis direction and showing the time period corresponding 
to the shutter period shown split. 

Fig. 62 shows a modified structure of the motion blurring adjustment unit 106. 
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Fig.63 is a flowchart for illustrating the processing for adjusting the amount of 
motion blurring contained in the foreground component picture 

Fig.64 is a block diagram showing a modified structure of the function of a 
signal processor 12. 

Fig. 65 shows the structure of a synthesis unit 371. 

Fig. 66 is a block diagram showing another modified structure of the function 
of the signal processor 12. 

Fig.67 is a block diagram showing the structure of a mixing ratio calculating 
unit 401. 

Fig.68 is a block diagram showing the structure of a foreground/background 
separating unit 402. 

Fig. 69 is a block diagram showing a further modified structure of the function 
of the signal processor 12. 

Fig. 70 illustrates the structure of a synthesis unit 43 1 . 

Fig. 71 shows another illustrative structure of a signal processing apparatus 
according to the present invention. 

Fig.72 is a flowchart for illustrating the processing for adjusting the amount of 
motion blurring by a signal processor 452. 

Fig. 73 shows an illustrative structure of a signal processing apparatus according 
to the present invention. 

Fig.74 shows a structure of a pressure area sensor 501. 
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Fig.75 illustrates the load applied to the pressure area sensor 501. 
Fig.76 illustrates typical weight data output by the pressure area sensor 501. 
Fig. 77 is a flowchart for illustrating the load calculating processing executed by 
a signal processor 502. 

Fig.78 is a block diagram showing the structure of generating a picture having 
an increased number of pixels per frame, as another function of the signal processor 
12. 

Fig. 79 illustrates pixel arrangement and an area corresponding to a pixel 
doubled in horizontal density. 

Fig.80 illustrates a picture component of a picture corresponding to light input 
to areas A to r. 

Figs.81A> 81B, 81C and 81D illustrates calculation of picture components 
corresponding to two areas of a pixel. 

Fig. 82 shows a typical input picture. 
Fig. 83 shows a typical double horizontal density picture. 
Fig. 84 shows a typical double vertical density picture. 
Fig. 85 shows a double density picture. 

Fig.86 is a flowchart for illustrating the processing for generating a double 
density picture by a signal processor 12 shown in Fig.78. 

Best mode for Carrying out the Invention 
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Fig. I shows the principle of the present invention. As may be seen in Fig. 1, a 
first signal, as the information of a real world 1 having the spatial axis and the 
temporal axis, is acquired by a sensor 2, and is made into data. A detection signal, as 
data 3 acquired by the sensor 2, is the information obtained on projecting the 
information of the real world 1 on a time space of a lower dimension than in the real 
world 1. Therefore, the information, resultmg from me projection, contamsdisto^^ 
ascribable to projection. Stated differently, the data 3, output by the sensor 2, is 
distorted relative to the information of the real world 1. Moreover, the data 3, thus 
distorted as a result of projection, also includes the significant information usable or 
correcting the distortion. 

Thus, according to the present invention, the data output by the sensor 2 is 
processed by a signal processor 4, whereby the distortion is removed, reduced or 
adjusted. Alternatively, the data output by the sensor 2 is processed by the signal 
processor 4 to extract the significant information. 

Fig.2 shows an illustrative structure of a signal processing apparatus according 
to the present invention The sensor 1 is comprised e.g., of a video camera, which 
photographs a picture ofthe real world to output the resulting picture data to the signal 
processor 12. The signal processor is comprised e.g., of a personal computer for 
processing the data input from the sensor 11, adjusting the amount of distortion 
produced by projection, specifying an area containing the significant information 
buried by the projection, extracting the significant information from a specified area 
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and for processing the input data based on the extracted significant information. 

The significant information may, for example, be a mixing ratio, as later 
explained. 

Meanwhile, the information indicating an area containing the significant 
information buried by the projection may also be deemed to be the significant 
information. Here, the areal information, as later explained, corresponds to the 
significant information. 

The signal processor 12 is configured as shown for example in Fig.3. A CPU 
(central Processmgunit) 21 executes various processmg operations m accordance with 
a program stored in a ROM (read-only memory) 22 or in a storage unit 28 . In a RAM 
(random access memory) 23, the program executed by the CPU 2 1 or data are stored 

as necessary. The CPU 21, ROM 22 and the RAM 23 are interconnected over a bus 
24. 

To the CPU 2 1 is connected an input/output interface 25 over a bus 24. To the 
input/output interface 25 are connected an input unit 26, comprised of a keyboard, a 
mouse and a microphone, and an output unit 27, comprised of a display and a speaker. 
The CPU 21 executes various processing operations responsive to commands input 
from the input unit 26. The CPU 21 outputs a picture, speech and so forth, obtained 
on processing, to the output unit 27. 

The storage unit 28, connected to the input/output interface 25, is constituted 
e.g., by a hard disc, for storing the program executed by the CPU 2 1 and a variety of 
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data. Acommunicationunit29conuTiunicate with external equipment overanetwork, 
such as Internet. In the present embodiment, the communication unit 29 operates for 
acquiring an output of the sensor 11. 

The program may also be acquired over the communication unit 29 for storage 
in the storage unit 28. 

A driver 30 connected to the input/output interface 25 drives a magnetic disc 
51, an optical disc 52, a magneto-optical disc 53 or a semiconductor memory 54, to 
acquire the program and data recorded therein, when these devices are connected 
thereto. The program and the data, thus acquired, are transferred to the storage unit 
28, as necessary, for storage therein. 

Referring to the flowchart of Fig.4, the operation performed by the signal 
processing apparatus based on the program stored in the storage unit 28, is explained: 
First, at step S 1 , a picture of an object, acquired by the sensor 1 1 , is acquired through 
e.g., the communication unit 29. The CPU 21 of the signal processor 12 sends the 
acquired picture data to the storage unit 28 for storage therein. 

Fig.5 shows a picture associated with the so-acquired picture data. The picture, 
shown in this embodiment, is comprised of a foreground 62 arranged ahead of a 
background 61 . The foreground here is a toy plane moving at a predetermined speed 
ahead of the still background 6 1 towards right in the drawing. The result is that the 
picture of the foreground 62 is a picture subjected to so-called motion blurring. 
Conversely, the picture of the background 61 is stationary and hence is a clear picture 
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free of motion blurring. A mixed area 63 is a picture comprised of a mixture of an 
object which is the background 61 and an object which is the foreground 62. 

Then, at step S2 9 the CPU 21 detects the mixed area of the objects. In the 
embodiment of Fig.5, the mixed area 63 is detected as an area of the mixture of the 
two objects. 

The CPU 21 at step S3 decides whether or not the objects are mixed. If the 
objects are not mixed, that is if there is no mixed area 63, the picture is not what is to 
be processed by the present information processing apparatus and hence the 
processing is finished. 

If conversely a decision is made at step S3 that the objects are mixed, the CPU 
2 1 proceeds to step S4 to find the object mixing ratio in the detected mixed area. The 
mixing ratio may be found by finding the motion vector of the foreground 2 1 relative 
to the background 61 and by fitting, from the motion vector, so that the mixing ratio 
in the mixed area 63 will be changed in a range from 0 to 1. At step S5, the CPU 21 
performs the processing of separating the objects in the mixed area 63 where plural 
objects are mixed together, based on the so-found mixing ratio. 

The above-described processing is explained in further detail, taking a picture 
of Fig.5 as an example. If picture data on one line of a portion 63A on the right end 
of the mixed area 63 of Fig.5 is plotted, the result is as shown in Fig.6, in which the 
abscissa denotes X-coordinates (coordinates in the horizontal direction in Fig.5) and 
the ordinate denotes pixel values on the X-coordinates. 
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A curve LI denotes pixel values on a line of a first timing, whilst a curve L2 
denotes pixel values on another line of the next timing. Similarly, curves Light 
reflecting layer 3 and L4 denote pixel values of lines of the sequentially consecutive 
timings. Stated differently, Fig.6 shows changes in the pixel values on associated lines 
at the four consecutive timings. 

The curve Li shows the state in the first timing in which state the foreground 62 
has not yet been imaged. So, the curve LI represents pixels of the foreground 61. 

On the curve LI, the pixel value is approximately 75 in the vicinity of the X- 
coordinate 140, and is increased to approximately 130 at the X-coordinate 145. The 
pixel value then is lowered and is approximately 120 in the vicinity of the X- 
coordinate 149. As the X-coordinate is increased, the pixel value is again increased 
and reaches approximately 160 in the vicinity of the X-coordinate 154. The pixel 
value then is again lowered and reaches approximately 130 in the vicinity of the X- 
coordinate 162. Then, in the vicinity of the X-coordinate of 165, the pixel value is 
approximately 180 and, in the vicinity of the X-coordinate of 170, the pixel value is 
again lowered to approximately 125. Then, in the vicinity of the X-coordinate of 172, 
the pixel value is increased to approximately 175 and, in the vicinity of the X- 
coordinate of 178, the pixel value is lowered to approximately 60. Subsequently, the 
pixel value is slightly fluctuated between 60 and 80 in a domain of the X-coordinates 
of from 178 to 195. In the X-coordinates on the further right side of approximately 
195, the pixel value is again increased to approximately 160. 
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As for the curve L2 of the next frame, the pixel value is constant at 
approximately 200 up to the pixel value of 145. The pixel value then is gradually 
lowered in a range from the X-coordinate of 145 to the Y-coordinate of 160, at which 
Y-coordinate value the pixel value is approximately 125. The curve then undergoes 
changes in a manner similar to those of the curve LI. 

The pixel value of the curve Light reflecting layer 3 is substantially constant at 
200 up to the vicinity of the X-coordinate 158 and is then lowered to approximately 
164 at the X-coordinate 164, after which it is increased to approximately 190. The 
curve then undergoes changes in a manner similar to those of the curve LI . 

The pixel value of the curve L4 is constant at approximately 200 from the 
vicinity of the X-coordinate of 140 up to the vicinity of the X-coordinate 170, and is 
abruptly lowered from the vicinity of the X-coordinate of 170 up to the vicinity of the 
X-coordinate 180, with the pixel value in the vicinity of the X-coordinate of 170 being 
approximately 70. The curve then undergoes changes in a manner similar to those of 
the curve LI. 

These changes in the pixel values of the curves L2 to L4 are ascribable to the 
fact that, while the picture of only the background 61 exists in the state of the curve 
LI, the picture of the foreground 62 is gradually increased with the movement of the 
picture of the foreground 62, that is with lapse of time. 

Specifically, as may be seen from comparison of the curve L 1 and the curve L2 
of the next following timing, the values of the curves L2 to L4 are substantially equal 
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in values „p to the -inity ofthe X-coordinate of 147. Banning fro^ 
the X-coordinate 1 47, the values of the curve L2 differ from those of the curves Light 
reflecting layer 3, L4,becoming equal to the values ofthe curve U in vicinity ofthe 
X-coordinate 159. Subsequently, the pixel values ofthe curve L2 are approbate* 
equal to those in the curve LI. That is, the values ofthe curve L2 in an area Rl 
corresponding to a domain Dl from an X-coordinate 146 to an X-coordinate 159 
indicate that the foremost part ofthe foreground 62 has been moved from the left end 
to the right end of the domain Dl during one unit period. 

Similarly, the pixel values ofthe curve Light reflecting layer 3 ofthe next 
timing in an area Rigid substrate 2 corresponding to a domain D2 from an X- 
coordinate 159 to an X-coordinate 172 indicate that the foremost part of the 
foreground 62 has been moved in the interim. The pixel values ofthe curve L4 ofthe 
next timing in an area R3 corresponding to a domain D3 from the X-coordinate 172 
to an X-coordinate 184 indicate that the foremost part ofthe foreground 62 has been 
moved in the interim. 

So, if the pixel values of the curve LI, weighted on the basis of a mixing ratio 
ofthe foreground 62 to the background 61, are subtracted from the pixel values ofthe 
curve L2, a curve Lll shown in Fig7 is obtained. This curve Lll, tantamount to 
subtraction of the values corresponding to background 61 from the pixels of the 
foreground 62 in the mixed area 63, represents a picture ofthe foreground on the 
background having the pixel value of 0. Meanwhile, in Fig.7, the abscissa and the 
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pixels b3 to b8 of the background 61, with the pixels Al to A6 of the foreground 62 
moving rightwards at the timing t2 by one pixel, that is to the position of the pixels b4 
to b9 of the background 61. 

In similar manner, the pixels Al to A6 of the foreground 62 are sequentially 
moved rightwards at a pitch of one pixel as time elapses from timing t3 to timing t5. 
In this case, the pixel values yl to yf, obtained on averaging the pixels of the 
respective lines of the timings tl to t5, constitute pixels obtained on imaging, that is 
pixels exhibiting motion blurring, with the values being represented by the following 
equations: 

f 5 3 
y 4 = -.(a l + a 2 )+-.b 4 

\ 5 2 ""(2) 

X 5 i '"(3) 

y 6 = T- + a 2 + a 3 + --b 6 

1 5 - (4) 



1 

J , i s j 1 (a.+« 1 +fl, + « 4 + fl,+ a 6 ) 



"' (5) 
- (6) 



1 i 

y 9 = T- 0 3 + <** + a s + a 6 ) + -- b 9 

5 — (7) 

1,2 

3 5 —(8) 

1.3 V 



y b = 7-(«s + ««)+T-* fr 
I 4 . 5 



... (9) 

-(10) 

Meanwhile, yl, y2, yd, ye and yf are equal to background pixels bl, b2, bd, be 



c 5 6 5 ' 
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and bf, respectively. 

If pixels bl to bf of the background are removed, the background 61 and the 
foreground 62 in the mixed area 63 can be separated from each other. That is, plural 
objects can be separated from one another. Moreover, the background pixels b 1 to bf 
can be found by solving the above equations, using, for example, the least square 
method, by assuming the background pixels bl to bf to be known such as by 
employing the pixel values of the fore and aft shutter time (frame). This gives a 
foreground picture freed of the motion blurring. In this manner, distortion caused by 
projection in the information of the real world can be reduced to create a clear picture 
such as by processing for resolution creation. 

In Fig.4, it is the deterministic processing that is executed, that is, the previous 
processing is used as basis and the next following processing is executed on the 
assumption that the result of the previous processing is just. Alternatively, statistic 
processing is also possible, as now explained with reference to illustrative processing 
shown in Fig. 9. 

Specifically, when carrying out the statistic processing, the CPU 21 acquires 
picture data at step S2 1. This processing is similar to that performed at step SI in 
Fig.4. 

Next, st step S22, the CPU 21 performs the processing of finding the mixing 
ratio of the foreground an the background from the picture data obtained at step S2 1 . 
At step S23, the CPU 2 1 executes the processing of separating the foreground and the 
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background based on the mixing ratio found at step S22. 

If the statistic processing is used, the processing of deciding whether or not the 
boundary of an object exists, such as that at step S23 of Fig.4, is unnecessary, thus 
enabling the foreground and the background to be separated from each other more 
expeditiously. 

The foregoing shows the manner as to how a clear picture of the foreground 62 
can be separated and extracted from the motion-blurred picture obtained on 
photographing a picture of the foreground 62 moving ahead of the background 6 1 

A more specified embodiment of a signal processing apparatus for identifying 
an area having the significant information buried therein or extracting the so-buried 
significant information from data acquired from the sensor by the deterministic 
processing is now explained. In the following embodiment, a CCD line sensor or a 
CCR area sensor corresponds to the sensor, while the areal information or the mixing 
ratio corresponds to the significant information and the mixing of the foreground and 
the background or the motion blurring corresponds to distortion. 
Fig. 10 is a block diagram showing the signal processor 12. 
Meanwhile, it does not matter whether the respective functions of the signal 
processor 12 are to be implemented by hardware or by software. That is, the block 
diagrams of the present specification may be deemed to be a hardware block diagram 
or a functional software block diagram. 

It is noted that the motion blurring means distortion contained in a moving 
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object, which distortion is produced by the movement of an object in the real world 
being imaged and by imaging characteristics proper to the sensor 1 1 . 

In the present specification, the picture corresponding to an object in the real 
world is called a picture object. 

An input picture, supplied to the signal processor 12, is furnished to an object 
extraction unit 101, an area specifying unit 103, a mixing ratio calculating unit 104 and 
a foreground background separating unit 105. 

The object extraction unit 101 roughly extracts a picture object corresponding 
to a foreground object contained in the input picture to send the extracted picture 
object to a motion detection unit 102. The object extraction unit 101 detects the 
contour of the picture object corresponding to the foreground object contained in the 
input picture to roughly extract the picture object corresponding to the foreground 
object. 

The object extraction unit 101 roughly extracts the picture object corresponding 
to the foreground object contained in the input picture to route the extracted picture 
object to the motion detection unit 102. The object extraction unit 101 roughly 
extracts the picture object corresponding to the background object, based on the 
difference between the input picture and the picture object corresponding to the 
extracted foreground object. 

It is also possible for the object extraction unit 101 to roughly extract the 
picture object corresponding to the foreground object and the picture object 
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corresponding to the background object based on the difference between the 
background picture stored in an internal background memory and the input picture. 

The motion detection unit 102 computes the motion vector of the picture object 
corresponding to the roughly extracted foreground,, by techniques such as block 
matching method, gradient method, phase correlation method or the Pel-Recursive 
method, to route me motion vector so calculated and the position information of the 
motion vector (the information specifying the position of the pixel corresponding to 
the motion vector) to the motion blurring adjustment unit 106. 

In the motion vector, output by the motion detection unit 1 02, there is contained 
the information corresponding to a movement quantity v. 

It is also possible for the motion detection unit 102 to output the picture object 
based motion vector, along with the pixel position information specifying the pixels 
for the picture object, to the motion blurring adjustment unit 106. 

The movement quantity vis a value for representing position changes of picture 
corresponding to a moving object in terms of a pixel-to-pixel interval as unit. For 
example, if a picture of an object corresponding to the foreground is moved so as to 
be displayed at a position offset by four pixels in a frame with respect to a directly 
previous frame, the movement quantity v of the object corresponding to the 
foreground is 4. 

Meanwhile, the object extraction unit 101 and the motion detection unit 102 are 
used when the quantity of the motion blurring associated with a moving object is 
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adjusted in the motion blurring adjustment unit 106. 

The area specifying unit 103 sends the information specifying each pixel of an 
input picture to one of the foreground area, a background area or a mixed area and for 
indicating to which of the foreground area, background area and the mixed area belong 
the pixels, from pixel to pixel, to the mixing ratio calculating unit 104, 
foregrounds^ 

106. The aforementioned information is referred to below as the area information. 

The mixing ratio calculating unit 104 calculates the mixing ratio for pixels 
contained in the mixed area 63, based on the input picture and the area information 
supplied from the area specifying unit 103, to route the so-calculated mixing ratio to 

the fo^eground^ackgrolmdseparating unit 105. This mixing ratio is referred to below 
as a mixing ratio a. 

Themixing ratio a indicates the proportion in the pixel value of the components 
of a picture corresponding to the background object, as indicated by an equation (13) 
to be described later. These components are also referred to below as the background 
components. 

The foreground/background separating unit 105 separates the inputpicture into 
a foreground component picture, made up only of a picture component associated with 
the foreground, also referred to below as foreground components, and a background 
component picture, composed only of background components, based on the area 
information supplied from the area specifying unit 103, and on the mixing ratio a 
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supplied from the mixing ratio calculatingunit 104, to route the foreground component 
picture to the motion blurring adjustment unit 1 06 and to the selection unit 1 07. The 
separated foreground component picture may also be an ultimate output. It is possible 
to realize the foreground and the background more accurate than those obtained in the 

conventional systeminwhichonlymeforegroundandthebackground can be specified 
without taking the conventional mixed area into consideration. 

The motion blurring adjustment unit 106 decides a processing unit, indicating 
one or more pixels contained in the foreground component picture, based on the 
movement quantity v as found from the motion vector and on the area information. 
The processing unit is data for specifying a set of pixels to be processed for adjusting 
the quantity of the motion blurring. 

The motion blurring adjustment unit 106 adjusts the quantity of the motion 
blurring contained in the foreground component picture, such as by removing the 
motion blurring contained in the foreground component picture, decreasing the 
quantity of the motion blurring or increasing the quantity ofthe motion blurring, based 
on the motion blurring adjusting quantity input to the signal processor 12, foreground 
component picture supplied from the foreground/background separating unit 105, the 
motion vector supplied from the motion detection unit 102, along with the 
corresponding position mformation, and on the processing unit, to output the 
foreground component picture, adjusted for the quantity ofthe motion blurring, to the 
selection unit 107. The motion vector with its position information may not be used, 
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if so desired. 



The selection unit 107 selects one of the foreground componentpicture supplied 
from the foregrounoTbackground separating unit 105 and the foreground component 
picture from the motion blurring adjustment unit 106, adjusted as to the motion 
blurring quantity, to output the selected foreground component picture. 

Referring to Figs. 1 1 to 26, an input picture sent to the signal processor 12 is 
explained. 

Fig. 1 1 illustrates imaging by a sensor 1 1 constituted by a CCD video camera 
provided with a CCD (charge coupled device) which is a solid state imaging device. 
An object corresponding to the foreground in the real world is moved between the 
object of the background in the real world an the sensor 1 1 e.g., horizontally from left 
to right. 

The sensor 1 1 images an object corresponding to the foreground along with the 
object corresponding to the background. The sensor 1 1 outputs the photographed 
picture on the frame basis. For example, the sensor 1 1 outputs a picture of 30 frames 
per sec. The exposure time of the sensor 1 1 may be set to 1/30 sec. The exposure time 
is the time which elapses since the start of conversion of light input to the sensor 1 1 
into electrical charges until the end of the conversion of the input light into electrical 
charges. This exposure time is sometimes referred to below as the shutter time. 

Referring to Fig. 12, showingpixel arrangement, A to I denote individual pixels. 
The pixels are arranged in a plane corresponding to a picture. A detection element 
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associated with one pixel is arranged on the sensor 11. When the sensor 11 
photographs a picture, one detection element outputs apixel value associated with one 
pixel belonging to the picture. For example, the position of the detection device along 
the X-direction corresponds to the position on the picture in the transverse direction, 
whilst that along the Y-direction corresponds to the position on the picture in the 
longitudinal direction. 

Referring to Fig. 13, a detection device, such as the CCD, converts the input 
light into electrical charges, during the time corresponding to the shutter time, to store 
the as-converted electrical charges. The quantity of the electrical charges is 
approximately equal to the intensity of the input light and to the time during which the 
light is input. The detection device sums the electrical charges, converted from the 
input light, to the electrical charges, already stored, during the time corresponding to 
the shutter time. That is, the detection device integrates the input light during the time 
corresponding to the shutter time to accumulate electrical charges in an amount 
corresponding to the integrated light. The detection device is said to have an 
integrating effect with respect to time. 

The charges accumulated in the detection device are converted into an electrical 
voltage by a circuit, not shown. The voltage, in turn, is converted into a pixel value, 
such as digital data, which is output. So, the individual pixels, output by the sensor 1 1 , 
are of a value mapped to a one-dimensional space, which is the result of integration 
with respect to the shutter time of a spatially extended portion of an object 
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corresponding to the foreground or the background. 

By such accumulating operation of the sensor 11, the signal processor 12 
extracts the significant information buried in the output signal, such as the mixing ratio 
a. The signal processor 12 adjusts the quantity of distortion caused by the mixing of 
no other than the foreground picture object, for example, the quantity of the morion 
blurring. The signal processor 12 also adjusts the quantity of the distortion produced 
by the mixing of the foreground picture object with the background picture object. 

Fig. 14 illustrates a picture obtained on imaging an object corresponding to a 
moving foreground and an object corresponding to a still background. Fig. 14 A shows 
a picture obtained on imaging an object corresponding to the moving foreground and 
an object corresponding to the still background. In an embodiment shown in Fig. 14A, 
the object corresponding to the foreground is moving horizontally from left towards 
right relative to the picture. 

Fig.l4B is a diagrammatic view showing pixel values, corresponding to a line 
of the picture shown in Fig.l4A, as extended along the time axis. The transverse 
direction of Fig. 14B corresponds to the spatial direction X of Fig.HA. 

The pixels of the background area are constituted solely by the background 
components, that is components of a picture corresponding to a background object. 
The pixels of the foreground area are constituted solely by the foreground 
components, that is components of a picture corresponding to a foreground. 

The pixels of the mixed area are constituted from the background and 
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foreground components. The mixed area, the pixel values of which are constituted 
from the background components and the foreground components, may be said to be 
a distorted area. The mixed area is further classified into a covered background area 
and an uncovered background area. 

The covered background area is a portion of the mixed area in register with the 
foremost part along the proceeding direction of the foreground and is an area in which 
the background component is hidden by the foreground with lapse of time. 

On the other hand, the uncovered background area is a portion of the mixed 
area in register with the rear part along the proceeding direction of the foreground and 
is an area in which the background component presents itself with lapse of time. 

A picture comprised of the foreground area, background area, a covered 
background area or the uncovered background area is input as an input picture to the 
area specifying unit 103, mixing ratio calculating unit 104 and to the 
foreground/background separating unit 105. 

Fig. 15 illustrates the background area, foreground area, mixed area, covered 
background area and the uncovered background area, as described above. In relation 
to the picture shown in Fig. 14, the background area is a still portion, the foreground 
area is a moving portion, the covered background area of the mixed area is an area 
where the picture is changed from the background to the foreground, and the 
uncovered background area of the mixed area is an area where the picture is changed 
from the foreground to the background. 
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Fig. 16 diagrammaticaUy shows pixel values of neighboring pixels in a row in 
a photographed picture of an object corresponding to a still foreground and an object 
corresponding to a sill background, with the pixel values shown developed along the 
temporal axis direction. As the neighboring pixels, arranged in a row, it is possible to 
select pixels arranged on a line of a picture. 

The pixel values of F01 to F04, shown in Fig.16, are those of pixels of the 
object of the still foreground. The pixel values of B01 to B04, shown in Fig. 16, are 
those of pixels of the object of the still background. 

In Fig. 16, time elapses from above towards below. The position of an upper 
side of a rectangle in Fig.16 corresponds to the time the sensor 1 1 begins converting 
the input light into electrical charges, while that of the rectangle in Fig. 1 6 corresponds 
to the time the sensor 11 finishes the conversion of the input light into electrical 
charges. That is, the distance from the upper to the lower sides of the rectangle of 
Fig.16 corresponds to the shutter time. 

In the following description, it is assumed that the shutter time is equal to the 
frame interval. 

The transverse direction in Fig.16 corresponds to the spatial direction X, 
explained with reference to Fig. 14. More specifically, the distance from the left side 
of a rectangle "F01" to the right side of a rectangle "B04" in Fig; 16 is eight times the 
pixel pitch, that is the span of the eight consecutive pixels. 

If the foreground and the background object are still, the light input to the 
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sensor 1 1 is not changed during the time corresponding to the shutter rime. 

The time span corresponding to the shutter time is split into two or more equal 
time periods. For example, if the number of times of the virtual splitting is four, the 
diagram of Fig. 16 may be represented as the diagram of Fig. 17. The number of times 
of the virtual splitting is set in association with e.g., the movement quantity v in the 
shutter time of the object corresponding to the foreground. For example, if the 
movement quantity v is four, the number of times of the virtual splitting is 4, with the 
time span corresponding to the shutter time being then split into four. 

The uppermost row in the drawing corresponds to the first split time period 
since the time of shutter opening. The second row corresponds to the second splittime 
period since the time of shutter opening. The third row corresponds to the third split 
time period since the time of shutter opening, whilst the fourth row corresponds to the 
fourth split time period since the rime of shutter opening. 

The shutter rime split in association with the movement quantity v is also called 
the shutter time/v hereinbelow. 

When the object corresponding to the foreground is at a standstill, the light 
input to the sensor 1 1 is not changed. So, the foreground component FOl/v is equal 
to the pixel value F01 divided by the number of times of the virtual splitting. 
Similarly, when the object corresponding to the foreground is at a standstill, the 
foreground component F02/v is equal to the pixel value F02 divided by the number of 
times of the virtual splitting, whilst the foreground component F03/v is equal to the 
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pixel value F03 divided by the number of times of the virtual splitting and the 
foreground component F04/v is equal to the pixel value F04 divided by the number of 
times of the virtual splitting. 

When the object corresponding to the background is at a standstill, the light 
incident on the sensor 1 1 is not changed. So, the background component BOl/v is 
equal to the pixel value B01 divided by the number of times of the virtual splitting. 
Similarly, when the object corresponding to the background is at a standstill, the 
background component B02/v is equal to the pixel value B02 divided by the number 
of times of the virtual splitting, whilst the background component B03/v is equal to the 
pixel value BOS divided by the number of times of the virtual splitting and the 
background component B04/v is equal to the pixel value B04 divided by the number 
of rimes of the virtual splitting. 

That is, when the object corresponding to the foreground is at a standstill, the 
light corresponding to the foreground input to the sensor 11 during the time 
corresponding to the shutter time remains unchanged. So, the first foreground 
component FOlv, corresponding to the shutter time/v, as from the shutter opening, the 
second foreground component FOlv, corresponding to the shutter time/v, as from the 
shutter opening, the third foreground component FOlv, corresponding to the shutter 
time/v, as from the shutter opening and the fourth foreground component FOlv, 
corresponding to the shutter time/v, as from the shutter opening, are of equal values. 
The above for FOI/v holds for F02/v to F04/v as well. 
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When the object corresponding to the background is at a standstill, the light 
corresponding to the background object input to the sensor II during the time 
corresponding to the shutter time remains unchanged. So, the first background 
component BOlv, corresponding to the shutter time/v, as from the shutter opening, the 
second background component BOlv, corresponding to the shutter time/v, as from the 
shutter opening, the third background component B0 1 v, corresponding to the shutter 
time/v, as from the shutter opening and the fourth background component BOlv, 
corresponding to the shutter time/v, as from the shutter opening, are of equal values. 
The above for BOl/v holds for B02/v to B04/v as well. 

In the following description, it is assumed that the object corresponding to the 
foreground is moving, with the object corresponding to the background being at a 
standstill. 

Fig. 18 diagrammatically shows pixel values of pixels arranged on a line 
including the covered background area when the object corresponding to the 
foreground is moving towards right in the drawing, with the pixel values being shown 
developed in the time axis direction. In Fig. 18, the movement quantity v of the 
foreground is 4. Since one frame is of short duration, it may be assumed that the 
object corresponding to the foreground is a rigid body and moving at an equal speed. 
In Fig. 18, the picture of the object corresponding to the foreground is moved so as to 
be displayed four pixels rightwards in a frame next to a direcdy previous reference 
frame. 
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In Fig.18, the leftmost to fourth left pixels belong to the foreground area. In 
Fig. 1 8, fifth left to seventh left pixels in Fig. 18 belong to the mixed area which is the 
covered background area. In Fig. 1 8, the rightmost pixel belongs to the background 



area. 



Since the object corresponding to the foreground is moved to hide the object 
corresponding to the background, as time elapses, the components contained in the 
pixel values of the pixels belonging to the covered background area are switched from 
the background component picture to the foreground component picture at a certain 
time point of the time period corresponding to the shutter time. 

For example, the pixel value M, shown with a bold line frame in Fig. 18, is 
represented by the equation (11): 

M = B02/v + B02/v + F07/V + F06/v 

-(11). 

For example, the pixel value M, shown with a bold line frame in Fig.18, 
contains the background component corresponding to one shutter time/v and the 
foreground component corresponding to three shutter time/v, the mixing ratio a of the 
fifth left pixel is 1/4. The sixth left pixel contains the background component 
corresponding to two shutter time/v and the foreground component corresponding to 
two shutter time/v, so the mixing ratio a is 1/2. The seventh left pixel contains the 
background component corresponding to three shutter time/v and the foreground 
component corresponding to one shutter time/v, so the mixing ratio a is 3/4. 
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Since the object corresponding to the foreground is a rigid body, such that the 
foreground picture is moved at an equal speed so as to be displayed four pixels 
towards right in the next frame, the first foreground component F07/vof the fourth left 
pixel in Fig. 18, with the first shutter time/v since the time of shutter opening, is equal 
to the second foreground component of the fifth left pixel in Fig. 1 8 corresponding to 
the second shutter time/v since the time of shutter opening. Similarly, the foreground 
component F07/v is equal to the foreground component of the sixth left pixel in Fig. 1 8 
corresponding to the third shutter time/v since the time of shutter opening and to the 
foreground component of the seventh left pixel in Fig. 1 8 corresponding to the fourth 
shutter time/v since the time of shutter opening. 

Since the object corresponding to the foreground is a rigid body, such that the 
foreground picture is moved at an equal speed so as to be displayed four pixels 
towards right in the next frame, the first foreground component F06/v of the third left 
pixel in Fig. 18, with the first shutter time/v since the time of shutter opening, is equal 
to the second foreground component of the fourth left pixel in Fig. 18 corresponding 
to the second shutter time/v since the time of shutter opening. Similarly, the 
foreground component F06/v is equal to the foreground component of the fifth left 
pixel in Fig. 18 corresponding to the third shutter time/v since the time of shutter 
opening and to the foreground component of the sixth left pixel in Fig. 18 
corresponding to the fourth shutter time/v since the time of shutter opening. 

Since the object corresponding to the foreground is a rigid body, such that the 
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foreground picture is moved at an equal speed so as to be displayed four pixels 
towards right in the next frame, the first foreground component F05/v of the second 
left pixel in Fig. 18, with the first shutter time/v since the time of shutter opening, is 
equal to the third foreground component of the fourth left pixel in Fig. 18 
corresponding to the second shutter time/v since the time of shutter opening. 
Similarly, the foreground component F05/v is equal to the foreground component of 
the fourth left pixel in Fig. 18 corresponding to the third shutter time/v since the time 
of shutter opening and to the foreground component of the fifth left pixel in Fig. 18 
corresponding to the fourth shutter time/v since the time of shutter opening. 

Smce the object corresponding to the foreground is a rigid body, such that the 
foreground picture is moved at an equal speed so as to be displayed four pixels 
towards right in the next frame, the first foreground component F04/v of the leftmost 
pixel in Fig. 18, with the first shutter time/v since the time of shutter opening, is equal 
to the second foreground component of the second left pixel in Fig. 18 corresponding 
to the second shutter time/v since the time of shutter opening. Similarly, the 
foreground component F04/v is equal to the foreground component of the third left 
pixel in Fig. 18 corresponding to the third shutter time/v since the time of shutter 
opening and to the foreground component of the fourth left pixel in Fig. 18 
corresponding to the fourth shutter time/v since the time of shutter opening. 

The foreground area corresponding to the moving object thus contains the 
motion blurring and hence may be said to be a distorted area. 
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Fig. 19 diagrammatically shows pixel values of pixels on a line comprehending 
the uncovered background area in case the foreground is moving towards right in the 
drawing, with the pixels shown extended in the time axis direction. In Fig. 19, the 
movement quantity v of the foreground is 4. Since one frame is of short duration, it 
may be assumed that the object corresponding to the foreground is a rigid body and 
moving at an equal speed. In Fig. 19, the picture of the object corresponding to the 
foreground is moved so as to be displayed four pixels rightwards in a frame next to a 
directly previous frame. 

In Fig. 19, the leftmost to fourth left pixels belong to the background area. In 
Fig. 19, fifth left to seventh left pixels belong to the mixed area which is the covered 
background area. In Fig. 1 9, the rightmost pixel belongs to the background area. 

Since the object corresponding to the foreground which has hidden the object 
corresponding to the background is moved so as to be removed from a position ahead 
of the object corresponding to the background, as time elapses, die components 
contained in the pixel values of the pixels belonging to the covered background area 
are switched from the background component picture to the foreground component 
picture at a certain time point of the time period corresponding to the shutter time. 

For example, the pixel value M\ shown with a bold line frame in Fig. 18, is 
represented by the equation (12): 

M' = F02/v + FOl/v + B26/v + B26/v 

- (12). 
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For example, the fifth left pixel contains the background component 
corresponding to three shutter time/v and the foreground component corresponding 
to one shutter time/v, the mixing ratio a of the fifth left pixel is 3/4. The sixth left 
pixel contains the background component corresponding to two shutter time/v and the 
foreground component corresponding to two shutter time/v, so the mixing ratio a is 
1/2. The seventh left pixel contains the background component corresponding to one 
shutter time/v and the foreground component corresponding to three shutter time/v, 
so the mixing ratio a is 1/4. 
56. 

If the equations (1 1), (12) are generalized, the pixel value M is represented by 
the following equation (13): 

M = a -P+ £ Filv 

-(13) 

where a is a mixing ratio, B is a pixel value of the background and Fi/v is the 
foreground component. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving at an equal speed, with the movement quantity v being 4, the 
first foreground component FOl/v of the fifth left pixel in Fig. 19, with the first shutter 
time/v since the time of shutter opening, is equal to the second foreground component 
of the sixth left pixel in Fig. 19 corresponding to the second shutter time/v since the 
time of shutter opening. Similarly, the FOl/v is equal to the foreground component of 
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